Hands-on Exercises
On
Accuracy assessment of map products derived
from Earth Observaon data
for
SERVIR WA Consortium Members and other Stakeholders
involved in Service Design, Planning and Implementation
University of Ghana, Legon
10-12 July 2023
By
Olatunji S. ABOYEJI
aboyeji@afrigist.org
Department of Geographic Information Science
African Regional Institute for Geospatial Information Science and Technology
(AFRIGIST), Obafemi Awolowo University, Ile-Ife, Nigeria
Introduction
The increasing popularity of remote sensing applications in various scientific fields and themes
can be attributed to significant advancements in sensor technology, automated processing
capabilities, and the widespread availability of free or affordable continuous Earth surface
measurements. This surge in interest is driven by the pressing need for precise and spatially-
explicit information about the Earth's physical characteristics, which is vital for sustainable
development initiatives related to efficient resource utilization, disaster risk reduction, and
ecosystem monitoring and preservation.
However, such use of remotely sensed data requires reliable and quantitative accuracy reports to
support confidence in the information generated. Accuracy assessment and validation are essential
in remote sensing-based projects since decision-making or scientific analysis with data of
unknown or little accuracy will result in information with low reliability, error propagation effects,
and, subsequently, be of limited value.
This manual will walk you through the process of performing area estimation for land use/land
cover, whether for single-date or change detection classifications. We will focus on sample-based
approaches for area estimation, which are preferred over pixel-counting methods due to potential
errors in maps derived from land cover/land use classifications. These errors could be caused by
pixel mixing or noise in the input data.
Using pixel-counting methods can lead to biased estimates of area, without providing information
about whether the estimates are overestimates or underestimates. On the other hand, sample-based
approaches allow us to generate unbiased estimates of area while accounting for the errors
associated with your map.
The primary tool for these exercises is the System for Earth Observation Data Access, Processing,
& Analysis for Land Monitoring (SEPAL). It is a web-based cloud computing platform that
enables users to create image composites, process images, classify images, etc., within the
browser. SEPAL integrates with Collect Earth Online (CEO) and the Google Earth Engine (GEE).
These exercises were adapted from the manual on SEPAL-CEO Area Estimation, Release 3-1-
2021 prepared Dyson and Tenneson (2021).
Exercise 1: Mosaic generation (Landsat & Sentinel 2)
Objectives
• Learn how to create an image mosaic
• Become familiar with a variety of options for selecting dates, sensors, mosaicking and
download options.
• Create a cloud-free mosaic
Prerequisites: SEPAL account
Creating a mosaic for classification
1. If SEPAL is not already open, click to open SEPAL in your browser: https://sepal.io/ and
login.
2. Click on the Processing tab.
3. Then, click on Optical Mosaic.
4. When the Optical Mosaic tab opens, you will see an Area of Interest window in the lower
right-hand corner of your screen.
There are three ways to choose your area of interest. Bring up the menu by clicking the carrot to
the right of the window label.
a. Select Country/Province (the default)
b. Select from EE table
c. Draw a polygon
5. Navigate using the map a State of a country and either draw a polygon around it or draw a
polygon within the borders. A smaller polygon will export faster.
6. [Optional] You can add a Buffer to your mosaic. This will include an area around the
province of the specified size in your mosaic.
7. Click Next.
8. In the Date menu you can select the Year you are interested in or click on More
a) This interface allows you to refine the dates or seasons you are interested in.
b) You can select a target date (The date in which pixels in the mosaic should ideally
come from), as well as adjust the start and end date flags.
c) You can also include additional seasons from the past or the future by adjusting the
Past Seasons and Future Seasons slider. This will include additional years’ data of
the same dates specified. For example, if you’re interested in August 2015,
including one future season will also include data from August 2016. This is useful
if you’re interested in a specific time of year but there is significant cloud cover.
d) For this exercise, let’s create imagery for the dry season of 2019
• Select July 1 of 2019 as your target date (2019-07-01), and move your date
flags to May 1-September 30.
• Click Next.
9. Select the Data Sources (SRC) you’d like. Here, select the Landsat L8 & L8 T2 option.
The color of the label turns brown once it has been selected. Then click Done.
• L8 began operating in 2012 and is continuing to collect dataL7 began
operating in 2001, but has a scan-line error that can be problematic for dates
between 2005-present
• L4-5 TM, collected data from July 1982-May 2012
• Sentinel 2 A+B began operating in June 2015
10. SEPAL will load a preview of your data. By default it will show you where RGB band data
is available. You can click on the RGB image at the bottom to choose from other
combinations of bands or metadata
11. We’re now going to go through the scene selection process. This allows you to change
which specific images to include in your mosaic.
• You can change the scenes that are selected using the SCN button on the lower right
of the screen. You can use all scenes or select which are prioritized. You can revert any
changes by clicking on Use All Scenes and then Apply.
• Change the Scenes by selecting Select Scenes with Priority: Target Date.
12. Click Apply. The result should look like the below image.
Notice the collection of circles over the study area and that they are all populated with a zero.
These represent the locations of scenes in the study area and the numbers of images per scene that
are selected. The number is currently 0 because we haven’t selected the scenes yet.
13. Click the Auto-Select button to auto-select some scenes
14. You may set a minimum and maximum number of images per scene area that will be
selected. Increase the minimum to 2 and the maximum to 100. Click Select Scenes. If there
is only one scene for an area, that will be the only one selected despite the minimum.
15. You should now see imagery overlain with circles indicating how many scenes are
selected.
16. In the window that opens, you will see a list of selected scenes on the right side of the
screen. These are the images that will be added to the mosaic. There are three pieces of
information for each:
• Satellite (e.g., L8, L7, L5 or L4)
• Percent cloud cover
• Number of days from the target date
17. You can also change the composing method using the CMP button on the lower right.
• Notice that there are several additional options including shadow tolerance, haze
tolerance, NDVI importance, cloud masking and cloud buffering.
• For this exercise, we will leave these at their default settings.
• If you make changes, click Apply after you’re don
18. Now we’ll explore the Bands dropdown. Click on the Red Green Blue at the bottom of
the page
19. The dropdown menu below will appear.
• Select the NIR, RED, GREEN band combination. This band combination displays
vegetation as red, with darker reds indicating dense vegetation. Bare ground and urban
areas appear grey or tan, while water appears black. NIR stands for near infrared.
• Once selected, the preview will automatically show what the composite will look like.
• Use the scroll wheel on your mouse to zoom in to the mosaic and then click and drag
to pan around the image. This will help you assess the quality of the mosaic.
Using what you’ve learned, take some time to explore adjusting some of the input parameters and
examine the influence on the output. Once you have a composite you are happy with, the mosaic
can be downloaded.
For example, if you have too many clouds in your mosaic, then you may want to adjust some of
your settings or choose a different time of year when there is a lower likelihood of cloud cover.
The algorithm used to create this mosaic attempts to remove all cloud cover, but is not always
successful in doing so. Portions of clouds often remain in the mosaic.
Naming and Saving your Recipe and Mosaic
1. Now, we will name the ‘recipe’ for creating the mosaic and explore options for the recipe.
You will use this recipe when working with the classification or change detection tools, as well
as when loading SEPAL mosaics into SEPAL’s Collect Earth Online.
You can make the recipe easier to find by naming it. Click on the tab in the upper right and
type in a new name. For this example, use YourStudyArea_LS8_2019_Dry.
2. Now let’s explore options for the recipe. Click on the three lines in the upper right-hand corner.
You can Save the recipe (SEPAL will do this automatically on retrieval) so that it is available
later.
Now if you click on the three lines icon, you should see an additional option:
• You can Revert to old revision…
• You can also Duplicate the recipe.
• Finally, you can Export the recipe. This downloads a zip file with a JSON of your
mosaic specifications.
3. Finally, we will save the mosaic itself. This is called ‘retrieving’ the mosaic. This step is
necessary to perform analysis on the imagery.
4. To download this imagery mosaic to your SEPAL account, click the Retrieve button.
5. A window will appear with the following options:
Bands to Retrieve: select the desired bands you would like to include in the download.
6. When you have the desired bands selected, click Retrieve.
7. You will notice the Tasks icon is now spinning. If you click on it, you will see the data retrieval
is in process. This step will take some time.
8. You can close the recipe by clicking on ‘x’ at the top-right corner of the screen (tab containing
the name of your recipe).
Congratulations! You have successfully completed this exercise. You now know how to create
a Landsat mosaic using the many customizable parameters in SEPAL.
Exercise 2: Image classification
Objectives
i. Learn how to create a classification scheme for land cover land use classification
mapping.
ii. Create training data for your classes that can be used to train a machine learning
algorithm.
iii. Run SEPAL’s classification tool.
The primary objective of this exercise is to generate a land cover map for a specific date by
classifying a Landsat composite created from Landsat images. Image classification is a commonly
used technique for mapping land cover, which involves identifying the composition of the
landscape (e.g., grass, trees, water, impervious surfaces) and land use, which describes the human
systems' organization on the land (e.g., farms, cities, wilderness). Mastering image classification
is essential and requires practical experience. This exercise provides an opportunity to gain that
experience.
Initially, you will determine the land cover classes you wish to map and assess the level of
variability within each class. There are two main approaches to image classification: supervised
(utilizing human guidance and training data) and unsupervised (without human guidance). One
supervised classification method involves employing a Machine Learning algorithm. Machine
Learning algorithms use training data along with image values to learn how to classify pixels. By
manually collecting training data, these algorithms can train a classifier and apply the identified
relationships to classify the remaining pixels in the map. The selection of image values (e.g.,
NDVI, elevation) used to train a statistical model should be carefully considered and informed by
your understanding of the phenomenon you aim to classify (e.g., forest, water, other, clouds).
In this exercise, we will utilize the Random Forest supervised classification algorithm within the
SEPAL environment to generate a land cover map. The algorithm will be trained to predict land
cover using a reference dataset generated by the user. This reference dataset can be collected either
through field observations or by examining remotely sensed data sources such as aerial imagery.
The resulting model will then be applied across the entire landscape to create the land cover map.
Response design for classification
Prior to commencing the image classification process, several preparatory steps need to be
undertaken. These include creating a response design that provides detailed information about
each land cover/land use category you intend to classify, generating mosaics for your area of
interest, and gathering training data for the classification model. Once these steps are completed,
the classification process can be executed, and the results can be examined.
The workflow outlined in this exercise has been adapted from exercises and materials developed
by Dr. Pontus Olofsson, Christopher E. Holden, and Eric L. Bullock at the Boston Education in
Earth Observation Data Analysis in the Department of Earth & Environment, Boston University.
For further information on their materials and research, please visit their GitHub site at
https://github.com/beeoda.
Specifying the classification scheme
1. “Classification scheme” is the name used to describe the land cover and land use classes
adopted. It should cover all of the possible classes that occur in the area of interest. A
classification scheme with detailed definitions of the land cover and land use classes to share
with interpreters can be created. For example, the green and red categories have multiple sub-
categories, which might be multiple types of forest or crops or urban areas. You can also have
classification schemes that are all one level with no hierarchical component.
2. When creating your own decision tree, be sure to specify if your classification scheme was
derived from a template, including the IPCC (Intergovernmental Panel on Climate Change)
land-use categories, CLC (CORINE land cover), or LUCAS (land cover and land use,
landscape).
• If applicable, your classification scheme should be consistent with the national land cover
and land use definitions.
• In cases where the classification scheme definition is different from the national definition,
you will need to provide a reason.
• Create a detailed definition for each land cover and land use and change class included in
the classification scheme. We recommend you include measurable thresholds.
Finding your Earth Engine Asset
1. Navigate to https://code.earthengine.google.com/ and login.
2. Navigate to your Assets tab in the left-hand column.
3. Under Assets, look for the name of the mosaic you just exported.
4. Click on the mosaic name.
5. You will see a window with information about your mosaic pop up.
6. Click on the two overlapping box icon to copy your asset’s location.
Creating classification & training data collection
There are multiple pathways for collecting training data. Using desktop GIS, including QGIS and
ArcGIS, to create a layer of points is one common approach. Using GEE is another approach. You
can also use CEO to create a project of random points to identify. All of these pathways will create
.csv or an GEE table that you can import into SEPAL to use as your training data set.
In this exercise, we will use built-in reference data collection tool in SEPAL to collect training
data. Even if you use a .csv or GEE table in the future, this is a helpful feature to collect additional
training data points to help refine your model.
In this assignment, you will create training data points using high-resolution imagery, including
Planet NICFI data. These will be used to train the classifier in a supervised classification using
SEPAL’s random forests algorithm. The goal of training the classifier is to provide examples of
the variety of spectral signatures associated with each class in the map
Setting up your classification
1. In the Process menu, click the green plus symbol and select Classification.
2. Add your optical mosaic for classification:
a. Click + Add and choose either Saved Sepal Recipe or Earth Engine
Asset (recommended).
b. If you choose Saved Sepal Recipe, simply select your recipe.
c. If you choose Earth Engine Asset, enter the Earth Engine Asset ID for the mosaic. The
ID of your asset should look like “users/username/OsunEast_L8_2020_Dry”.
Remember that you can find the link to your Earth Engine Asset ID via Google Earth Engine’s
Asset tab.
d. Select bands: Blue, Green, Red, NIR, SWIR1, & SWIR2. You can add other bands as well
if you included them in your mosaic.
e. You can also include Derived bands by clicking on the green button on the lower left.
f. Click Apply, then click Next.
3. In the Legend menu, click + Add This will add a place for you to write your first-class label.
a. Choose colors for each class as you see fit.
b. Click Done.
Collect training data points
In most cases, it is ideal to collect a large amount of training data points for each class that capture
the variability within each class and cover the different areas of the study area. However, for this
exercise, you will only collect a small number of points: around 25 per class. When collecting data
points, make sure that your plot contains only the land cover class of interest (no plots with a
mixture of your land cover categories).
1. In the upper right corner of the map is a stack of three rectangles. If you mouse over this icon,
it says “Select layers to show.”
Available base layers include SEPAL (Minimal dark Sepal default layer), Google Satellite, and
Planet NICFI composites.
a. We will use the Planet NICFI composites for this example. The composites are available
in either RGB or false color infrared (CIR). Composites are available monthly after
September 2020 and for every 6 months prior back till 2015.
b. Drag and drop the recipe, the Planet NICFI composite (to train the classifier) and the
composite to be classified.
c. Select Close
d. Click on Enable Reference Data Collection
e. You can also select “Show labels” to enable labels that can help you orient yourself in the
landscape
f. With reference data collection enabled, you can start adding points to your map.
g. Use the scroll wheel on your mouse to zoom in to the study area. You can click-hold and
drag to pan around the map. Be careful though, as a single click will place a point on the
map.
h. If you accidentally add a point, you can delete it by clicking on the red Remove button.
i. If you need to modify classification of any of your data points, you can click on the point
to return to the classification (or delete) options.
j. After you collect your training data for Forest, you may see the classification preview
appear.
k. Once you are satisfied with training data points a class, move on to the next class until all
have been collected.
Classification using machine learning algorithms (Random Forests) in SEPAL
Add training data collected outside of sepal [Optional]
1. If you collected training data using QGIS, CEO, or another pathway, you will need to add the
Training Data we collected in Exercise 2.3 in the TRN tab.
a. Click on the green Add button.
• You can upload a CSV file. ii. Or you can select Earth Engine Table and enter the path to
your Earth Engine asset in the EE Table ID field.
b. Click Next.
c. For Location Type, select “X/Y” coordinate columns” or “GEOJSON Column”
depending on your data source. GEE assets will need the GEOJSON column option.
d. Click Next.
e. Leave the Row filter expression blank. For Class format, select “Single Column” or
“Column per class” as your data dictates.
f. In the Class Column field select the column name that is associated with the class.
g. Click Next.
2. Now you will be asked to confirm the link between the legend you input previously and your
classification. You should see a screen as follows. If you need to change anything, click the
green plus buttons. Otherwise, click Done, then click Close.
Review additional classification options
1. Click on AUX to examine the auxiliary data sources available for the classification.
a. Auxiliary inputs are optional layers which can be added to help aid the classification. There
are three additional sources available: Latitude - Includes the latitude of each pixel; Terrain
- Includes elevation of each pixel from SRTM data; Water - Includes information from the
JRC Global Surface water Mapping layers.
b. Click on Water and Terrain.
c. Click Apply.
2. Click on CLS to examine the classifier being used.
a. The default is a random forest with 25 trees.
b. Other options include classification and regression trees (CART), Naive Bayes, support
vector machine (SVM), minimum distance, and decision trees (requires a CSV).
c. Additional parameters for each of these can be specified by clicking on the More button
in the lower left.
d. For this example, we will use the default random forest with 25 trees.
3. If you turned off your classification preview previously to collect training data, now is the time
to turn it back on.
• Click on the “Select layers to show” icon.
4. Now we’ll save our classification output.
a. First, rename your classification by typing a new name in the tab.
b. Click Retrieve classification in the upper right-hand corner (cloud icon).
c. Choose 30 m resolution.
d. Select the Class, Class probability, all the classes.
e. Retrieve to your SEPAL Workspace.
You can also choose Google Earth Engine Asset if you would like to be able to share your results
or perform additional analysis in GEE. However, with this option, you will need to download your
map from GEE using the Export function.
f. Once the download begins, you will see the spinning wheel in the bottom left of the
webpage in Tasks. Click the spinning wheel to observe the progress of your export.
g. When complete, if you chose SEPAL workspace, the file will be in your SEPAL downloads
folder. (Browse > downloads > classification name folder). If you chose GEE Asset the
file will be in your GEE Assets.
Spend some time looking at your land cover classification map in order to understand if the results
make sense. We’ll do this in the classification window. This allows us to visualize the data and
collect additional training points if we find areas of poor classification. Other approaches not
covered here include visualizing the data in Google Earth Engine or in another program, such as
QGIS or ArcMAP.
With SEPAL you can examine your classification and collect additional training data to improve
the classification
Congratulations! You now know how to produce map classifications in SEPAL.
Exercise3: Sample design and stratification
Objective
• Generate a stratified random sample based on your image classification
Once you have either a land use/land cover (LULC) map or a change detection map, the next step
is to estimate the area within each LULC type or change type and the error associated with your
map. All maps have errors, for example model output errors from pixel mixing or input data noise.
Our objective is to create unbiased estimates of the area for each mapped category.
To do this, we will use sample-based estimations of area and error instead of ‘pixel counting’
approaches. Pixel counting approaches simply sum the area belonging to each different class.
However, this doesn’t account for classification errors – for example, the probability that a pixel
classified as wetland should be open water. Therefore, the pixel counting approach provides no
quantification of sampling errors and no assurance that estimates are unbiased or that uncertainties
are reduced (Stehman, 2005; GFOI, 2016).
Sample-based estimations of area and error create estimations of errors in pixel classification and
use this to inform estimations of area. Therefore, sample-based estimations are in keeping with
the IPCC General Guidelines (2006) that estimates should not be over- or under- estimates, and
that uncertainty should be reduced as much as practically possible.
Steps to sample-based estimation of area and accuracy:
• First, you will use the different classes in your LULC or change detection map to create a
stratified sampling design in SEPAL using the Stratified Area Estimator (SAE) - Design
tool.
• Then you will revisit your response design and labelling protocols to use with data
collection in CEO.
• Finally, you will use data generated in CEO to calculate the sample-based estimates in
SEPAL, using the Stratified Area Estimator- Analysis tool.
• This tool quantifies the agreement between the validation reference points and the map
product, providing information on how well the class locations were predicted by the
Random Forest classifier.
This process will provide two important outputs. First, you will have estimates of the area for each
LULC or change type. Second, you will have a table that describes the accuracy for each LULC
or change type (confusion matrix). These may be final products for your projects. However, if you
decide that your map is not accurate enough, this information can be fed back into the classification
or change detection algorithms to improve your model.
Sample design and stratification
With stratified random sampling, each class (e.g., land use, land cover, change type) is treated as
a strata. Then, a sample is randomly taken from each strata, either in proportion to area, in
proportion to expected variance, or in equal numbers across strata.
Uploading files to SEPAL
If your classification is not stored in SEPAL (for example, a classification in GEE), you will need
to upload it to SEPAL in order to use SEPAL’s stratified random sample tool.
1. For either approach, first select the purple wrench Apps button. If you have an existing tab
open, you may need to click the plus sign in the top right.
2. Choose the R Studio application. You may be prompted to enter your SEPAL username
and password to enter R Studio
a. This will open an instance of RStudio, an IDE for the R programming language.
b. You should see a ‘Files’ tab in the lower right window.
c. If not, you may need to adjust the window layout. To do this, move your mouse to
the right-hand side of the window where a four-way arrow will appear. Click and
drag your mouse to the left to reveal the right pane.
d. Click the Upload button that is located in the lower right side of the R Studio
interface (see below).
e. In the Upload Files window, click Choose File.
f. Navigate to the correct location on your drive, select your map and click Open.
g. Once you’ve selected this file, click OK to complete the upload (see below).
h. You will see your file appear in the list of files in the lower right-hand pane.
i. You may now close the RStudio instance by clicking the tab’s x
Creating a stratified random sample
A well-prepared sample can provide a robust estimate of the parameters of interest for the
population (percent forest cover, for example). The goal of a sample is to provide an unbiased
estimate of some population measure (e.g., proportion of area), with the smallest variance possible,
given constraints including resource availability. Two things to think about for sample design are:
do you have a probability-based sample design? That is, does every sample location have some
probability of being sampled? And second, is it geographically balanced? That is, are all regions
in the study area represented.
1. First, navigate to https://sepal.io/ and sign in.
2. Select the Apps button (purple wrench).
3. Type ‘stratified’ into the search bar or scroll through the different process apps to find
“Stratified Area Estimator–Design”
4. Select Stratified Area Estimator-Design. Note that loading the tool takes a few minutes.
5. When the tool loads properly, it will look like the image below. Read some of the information
on the Introduction page to acquaint yourself with the tool.
a. On the Introduction page, you can change the language from English to French or
Spanish.
b. The Description, Background, and “How to use the tool” panels provide more information
about the tool.
c. The Reference and Documents panel provides links to other information about stratified
sampling, such as REDD Compass.
6. The steps necessary to design the stratified area estimator are located on the left side of the
screen and they need to be completed sequentially from top to bottom.
7. Select Map input on the left side of the screen.
a. For this exercise, we’ll use the classification from the previous exercise. However, you can
substitute another classification, such as the change detection classification if you would
like.
b. In the Data type section, click Input.
c. In the Browse window that opens, navigate to the Module 2 dataset and select it. This may
be in your “downloads” folder for retrieved classifications. Select the .tif file.
d. Then click Select.
e. Note that the Output folder section shows you where in your SEPAL workspace all the
files generated from this Exercise will be saved.
f. Optionally, you can use a csv with your raster areas instead. We won’t discuss that here.
8. Next, click Strata areas on the left side of the screen.
9. In the Area calculation section, select OFT. OFT stands for the Open Foris Geospatial
Toolkit.
10. Click the Area calculation and legend generation button. This will take a few minutes to
run. After it completes, notice that it has updated the Legend labeling section of the page.
a. Next, you will need to adjust the class names in the Legend labeling section. Type in the
following class names in place of the numeric codes.
b. Now click Submit Legend. The Legend and Areas section will now be populated with
the map code, map area, and edited class name.
c. You can now Rename and Download the area file if you would like. However, it will save
automatically to your Sepal workspace.
11. When you’re done, click on Strata selection on the left panel.
12. Now you need to specify the expected accuracies. You will do this for each class.
a. You can get more information by clicking the plus button to the right of the box that
says What are the expected accuracies?
b. Specifying the expected user accuracy helps the program determine which classes might
need more points relative to their area.
c. Some classes are easier to identify–including common classes and classes with clear
identifiers like buildings.
d. Classes that are hard to identify include rare classes and classes that look very similar to
one another. Having more classes with low confidence will increase the sample size.
1. Select the value for classes with high expected user accuracy with the first slider. This is
set to 0.9 by default, and we’ll leave it there.
2. Then, select the value for classes with low expected user accuracy with the second
slider. This is set to 0.7 by default, and we’ll leave it there as well.
13. Now we need to assign each class to the high or the low expected user accuracy group.
a. For this exercise, please assign all the classes to the high confidence class. If you assign
any to the low confidence class, you will not be able to use the CEO-SEPAL bridge in
the next Exercise.
b. Then, click on the box under “low confidence” that appears and assign the
corresponding class(es).
c. If you make a mistake, there’s no way to remove the classes. However, just change one
of the sliders slightly, move it back, and the class assignments will have been reset.
d. DO NOT assign your No Data class to either high or low condence.
14. When you’re satisfied, click on Sampling Size on the left panel.
a. Now we will calculate the required sample size for each strata.
b. You can click on the “+” button to get more information.
c. First, we need to set the standard error of the expected overall accuracy. It is 0.01 by
default, however for this exercise we will set it to 0.02.
i. This value affects the number of samples placed in each map class. The lower the value,
the more points there are in the sample design. Test this by changing the error from
0.05 to 0.01, and then change it back to point 0.05.
ii. Note that you can adjust this incrementally with the up/down arrows on the right side
of the parameter.
d. Then determine the minimum sample size per strata. By default, it is 100. For the
purposes of this test, we will set it to 30.
e. You can also check the “Do you want to modify the sampling size” box.
f. If you would like, you can edit the name of the file & download a csv with the sample
design. The file contains the table shown above with some additional calculations.
However, SEPAL will automatically save this file.
15. When you’re ready, click on Sample allocation to the left.
a. The final step will select the random points to sample.
b. Select Generate sampling points and wait until the progress bar in the bottom right
finishes. Depending on your map, this may take multiple minutes. A map will pop up
showing the sample points. You can pan around or zoom in/out within the sample points
map.
i. The resulting distribution of samples should look similar to the below image. These
values will vary depending on your map and the standard error of expected overall
accuracy you set.
ii. Sometimes this step fails, no download button will appear, and you will need to refresh
the page and restart the process.
16. Now fill out the fields to the right.
a. Specify the number of operators, or people who will be doing the classification. Here,
leave it set to 1. For CEO, this might be the number of users you think your project will
have.
b. The size of the interpretation box depends on your data and corresponds to CEO’s
sample plot. This value should be set to the spatial resolution of the imagery you classified
(Landsat= 30 meters). Here, leave it at 30 m.
17. If you would like to create a project via CEO, click on Download as tabular data (.csv).
18. To create a project via the CEO-SEPAL bridge, click on Create CEO project.
a. This will create a CEO project via the CEO-SEPAL bridge.
b. This process will take a few minutes and you should see text and completion bars in the
lower right as calculations happen.
19. You can download a .shp file to examine your points in QGIS, ArcGIS, or another GIS
program. You can also create a CEO project using a .shp file, however that is outside of the
scope of this manual.
Creating a CEO project via CSV
1. Make sure you have downloaded the .csv of your stratified random sample plots.
2. Open your downloaded .csv file in Excel or the spreadsheet program of your choice.
3. First, make sure that your data doesn’t contain a strata of ‘no data’. This can occur if your
classification isn’t a perfect rectangle. If you have ‘no data’ rows, return to the SEPAL stratified
estimator, and be sure to not include your no data class in the strata selection step.
4. Right now, your stratification is grouped by land cover type (map_class column). To reduce
the human tendency to use the order of the plots to help identify them (i.e. knowing the first
100 plots were classified forest, so being more likely to verify them as forest instead of
determining if that is correct) we suggest first randomizing the order of the rows. This is
optional.
To do this, click the Sort & Filter button in Excel
5. Next, Sort on the ‘id’ field by value, either smallest to largest or largest to smallest.
6. Now we need to add the correct columns for CEO. Remember that Latitude is the Y axis and
longitude is the X axis. For CEO, the first three columns must be in the following order:
longitude, latitude, plotid. The spelling and order matter. If they are wrong CEO will not work
correctly.
a. Rename ‘id’ to PLOTID. You can also add a new PLOTID field by creating a new column
labeled PLOTID, and fill it with values 1-(number of rows).
b. Rename the ‘XCoordinate’ column to ‘LONG’ or ‘LONGITUDE’.
c. Rename the ‘YCoordinate’ column to ‘LAT’ or ‘LATITUDE’.
d. Reorder the columns in Excel so that LAT, LONG, PLOTID are the first three columns, in
that order.
7. Save your updated .csv, making sure you save it as a .csv and not as an .xlsx file.
8. Navigate to https://collect.earth.
a. Creating a project in CEO requires you to be the administrator of an institution.
b. Login to your CEO account. If you’re already the administrator of an institution, navigate
to your institution’s landing page by typing in the institution’s name and then clicking on
the Visit button.
c. If you’re not an admin, go ahead and create a new institution.
d. Click on create new institution from the homepage, then fill out the form & click create
institution.
9. When you’re on the institution’s page, click on the “Create New Project” button.
10. This will go to the Create Project interface. We’ll now talk about what each of the sections in
the wizard does. For more information, please see the Institutional Manual available on the
CEO Support page https://collect.earth/support.
a. EMPLATE: This section is used to copy all the information—including project info, area,
and sampling design—from an existing published project to a new project.
i. This is useful if you have an existing project you want to duplicate for another year or
location, or if you’re iterating through project design. You can use a published or closed
project from your institution or another institutions’ public project.
ii. The project id is found in the URL when you’re on the data collection page for the
project.
b. Enter the project’s Name and Description.
i. The Name should be short and will be displayed on the Home page as well as the
project’s Data Collection page.
ii. You should keep the Description short but informative.
iii. The Privacy Level radio button changes who can view your project, contribute to data
collection, and whether admins from your institution or others creating new projects
can use your project as a template.
iv. Click on any Project Options you would like.
c. Imagery Selection allows you to select any of the public or your institutional imagery.
The default public imagery includes MapBox and Planet NICFI data.
d. Under Plot design, the project area of interest (AOI) determines where sample plots will
be drawn from for your project. This is the first step in specifying a sampling design for
your project. There are four main approaches for specifying an AOI and sampling design
(see the image below). For this Exercise, we will use the csv file downloaded from SEPAL.
e. Click the dropdown arrow and select CSV file.
i. Click on Upload, and upload the .csv of your stratified random sample. Note that the
number of plots you want to sample must be 5000 or less.
ii. Select if you would like round or square plots, and specify the size. For example, you
might specify square plots of 30m width in order to match Landsat grid size.
iii. Under Assign plots, select No Assignment. This is for the purpose of this exercise. In
reality, you will have to assign plots as the Coordinator to many Users, and for quality
control checks, you will have to assign Subject Matter Expert (SME) users.
f. Sample Point Design: You need to determine the sample point design within each sample
plot.
a. You can choose Random or Gridded, and how many samples per plot or the sample
resolution respectively. You can also choose to have one central point.
b. Using CEO’s built in system, the maximum number of sample points per plot is 200.
The maximum total number of sample points for the project across all plots is 50,000.
g. Survey Design: This is where you design the questions that your data collectors/photo
interpreters will answer for each of your survey plots. Each question creates a column of
data. This raw data facilitates calculating key metrics and indicators and contributes to
fulfilling your project goals.
i. Survey Cards are the basic unit of organization. Each survey card creates a page of
questions on the Data Collection interface. You can preview your survey questions in
the right-hand pane.
ii. The basic workflow is: Create new top-level question (new survey card) THEN
populate answers THEN create any child questions & answers THEN move to next
top-level question (new survey card) & repeat until all questions have been asked.
iii. You can ask multiple types of questions (including the button—text questions from the
Simple interface). You can also add survey rules in the Survey Rules Design panel.
iv. Broadly, there are four question types and three data types. They are combined into 10
different component types.
v. The four question types are:
• Button: This creates clickable buttons, allowing users to select one out of many
answers for each sample point.
• Input: Allows users to enter answers in the box provided. The answer text provided by
the project creator becomes the default answer.
• Radio button: This creates radio buttons, allowing users to select one out of many
answers for each sample point.
• Dropdown: Allows users to select from a list of answers.
vi. The three data types allowed are:
• Boolean: Use this when you have two options for a question (yes/no).
• Text: Use this when you have multiple options which are text strings. They may include
letters, numbers, or symbols.
• Number: Use this when you have multiple options that are numbers, which do not
contain letters or symbols.
vii. First, type in your question in the New question box, such as “Is this forest or non-
forest?”
viii. Then click add survey question.
ix. A new survey card (Survey Card Number 1) will pop up with your question in it.
x. You can now add answers.
xi. Create one answer for each of your land use types. Here we will use 1 and 2 to match
our “Forest” and “Built-up” in our classification. Be sure to include all your land use
types.
xii. Note that the Stratified Area Estimator–Analysis only accepts numeric values for the
land use types. If you would like to use human-readable text values (e.g. Forest instead
of 1), you MUST follow the directions in the previous exercise “Preparing your
CEO collected data for analysis in SEPAL”.
xiii. You can add additional survey questions if you’d like to experiment. An example of
two survey cards is shown below.
11. When you’re done, click Create Project.
a. If you’re successful, you’ll see the review project pane.
b. The Project AOI will now show the location of a subset of your plots (a maximum number
can be displayed).
You have successfully created a stratified random sampling design for your map and a project
(CEO or CEO-SEPAL) to collect reference data.
Exercise 4: Reference Data collection and Area/uncertainty estimation
Objectives
i. Understand how to set up a successful verification project
ii. Collect land cover verification data about each of your sample points.
iii. Create quality management protocols for your verification project.
iv. Create area estimates for your classification
v. Create uncertainty/error estimates for your classification
Data collection
Once a stratified random sample has been created, the next step is to use CEO (or optionally the
CEO-SEPAL tool) to visually interpret the land cover at the sample locations using a suitable
source of reference data, often remote sensing data. These visual interpretations will then inform
the area and error estimation. However, to ensure accurate human interpretation of land cover, you
will need to adopt data quality management approaches. Thus, in this exercise, you will check
your classification design, plan your data collection and collect your data.
The reason for this focus on data quality is simple: area and error estimates are based on the human
interpreter’s labelling of the sample; therefore, it is important that the labels are correct. Some
recommend that three interpreters examine each unit independently, while other projects just have
a subsample of the data points cross checked by another interpreter.
Identifying the reference data sources
Ideally, you would have plots revisited in the field. However, this is rarely attainable given limited
resources. An alternative is to collect reference observations through careful examination of the
sample units using high resolution satellite data, or moderate resolution if high resolution is not
available. The more data you have at your disposal the better.
If you have no additional data, you can use remote sensing data, such as Landsat data, for
collecting reference observations, as long as the process to collect the reference data is more
accurate than the process used to create the map being evaluated. Careful manual examination can
be regarded as being a more accurate process than automated classification.
The CEO-SEPAL bridge uses only the default imagery, which is currently Mapbox Satellite.
Collecting data
After the training interpreters and samples allocated by the coordinator, data collection starts. This
can occur in the CEO-SEPAL interface (for smaller projects) or via CEO for larger or multi-user
projects.
Data collection in CEO
1. To collect data in CEO, navigate to the project you created earlier. Your screen should look
like this:
2. Click Go to first plot. This will take you to your first plot.
3. Answer all of the questions for your first plot by clicking on the appropriate answers.
a. If you created multiple questions, you can navigate between questions using the numbers
above your question text.
b. Scroll in and out with your mouse wheel (or press the +/- buttons) to view the landscape
context and see your plots properly.
c. Click on Save to save your answers and move on to the next plot.
4. Continue answering questions until you reach the last plot.
5. When you have finished answering all of the questions, navigate to your Institution’s page.
6. Your project name should now be green, indicating that all plots have been completed. If it is
yellow, click on the project name and answer the remaining questions
7. Click on the S next to the project.
8. This will download your project’s sample data. Save it to your hard drive.
Area and uncertainty estimation
The final step of calculating the sample-based estimates of error and area is taking the map areas,
and your verification data points from our data collection, conducted according to the response
design rules and using statistics to output the final estimates of area and uncertainty.
Understanding the error matrix
The error matrix organizes the acquired sample data in a way that summarizes key results and aids
the quantification of accuracy and area. This is a simple cross-tabulation that compares the
(algorithm assigned) map category labels to the (human assigned) reference category labels (your
validation classification). The count for each pairwise combination are included in the blue and
yellow cells in the following example.
• The main diagonal of the error matrix (blue cells) includes the count of the number of
correct classifications.
• The off-diagonal elements (yellow cells) show map classification errors.
• The user’s accuracy can be quantified by dividing the number of correctly classified plots
by the sum of the plots classified as the mapped class. For the forest class in the example
above, this is 17 correctly identified points divided by 19 total forest plots. User’s
accuracies for each class are shown in the orange cells. User’s accuracy is the complement
of errors of commission (sites that are classified as forest in the map, but are not actually
forest).
• The producer’s accuracy can be quantified by dividing the number of correctly classified
plots by the sum of the plots classified as the mapped class in the validation reference
sample. For the forest class in the example above, this is 17 correctly identified points
divided by 20 samples that were classified as forest from the reference data. Producer’s
accuracies for each class are shown in the pink cells. Producer’s accuracy is the
complement of errors of omission (sites that are not classified as forest in the map that are
actually forest).
Preparing your CEO collected data for analysis in SEPAL
1. Open the .csv file you downloaded from Collect Earth Online. It will probably have a name
like “ceo-project-name-sample-data-yyyy-mm-dd.csv”.
2. Inspect the column data. You should also have a column with your question about the
correct map class as the column header, e.g., Is this forest? Can we regard this as
herbaceous? etc. These are the classes you assigned manually in CEO based on map
imagery. This will either be numeric (1 or 2) or text (Yes and No) depending on how you
set up your Collect Earth Online project.
3. If your column for the correct map class is numeric, skip to step 5 below.
4. If your column for the correct map class is text, you will need to either:
a) Check that your text column matches exactly the Legend Labels you added during
sample design.
b) Check that capitalization is the same, e.g., Forest and not forest.
c) OR create another column with the associated numeric value.
i. First, create a new column and name it COLLECTED_CLASS.
ii. In the formula cell, type: =IF([text column letter]2=”Yes”,1,2). For this
example, the text column letter is U.
iii. This will use an if statement to assign the number 1 to sample plots you
assigned the value “Forest” to, and the number 2 to other plots (here, plots
labeled Non-forest). If you have more than two classes, you will need to
use nested IF statements.
iv. Press Enter. You should now see either a 1 or a 2 populate the column.
Double check that it is the correct value.
v. Fill the entire column.
5. Save your .csv file.
6. Upload your .csv file to SEPAL
Using the stratified estimator in SEPAL
1. Open the Stratified Area Estimator-Analysis Tool
a) In the Apps SEPAL window select Stratified Area Estimator - Analysis.
b) This tool is very similar to the Design tool that you used to create your stratified
sample.
i. You will land on the Introduction page which allows you to choose your
language and provides background information on the tool. Note that Reference
and Documents are in the same place as the Design tool.
ii. The pages that contain the necessary steps for the workflow are on the left side
of the screen and need to be completed sequentially.
2. Select the Inputs page on the left side of the screen. You will see two data requirements under
the Select input files section.
a. Reference Data this refers to the table that you classified and exported in the previous
section. It will contain a column that identifies the map output class for each point as well
as a column for the value from the image interpreter (validation classification).
• For projects completed in CEO: Select the Reference data button and navigate to the
.csv file you downloaded from CEO and then uploaded to SEPAL.
b. Area data this is a CSV that was automatically created during the Stratified Area
Estimator–Design workflow. It contains area values for each mapped land cover class.
i. Click the Area data button.
ii. Open the folder labeled sae_design_your-name-here.
iii. Within this folder, select area_rast.csv (see image below).
3. Next, you will need to adjust some parameters so that the tool recognizes the column names
for your reference data and area data that contain the necessary information for your accuracy
assessment. You should now see a populated Required input panel on the right side of the
screen.
a. Choose the column with the reference data information: COLLECTED_CLASS.
b. Choose the column with the map data information: PL_MAP_CLASS
c. Choose the map area column from the area file—map_area
d. Choose the class column from the area file—map_code or map_edited_class
i. The map_edited_class has the names you entered manually during the design phase,
while the map_code has the numeric class codes.
ii. For projects completed in CEO: Use map_code if you have a column in your reference
data. If you use map_edited_class you must make sure that capitalization is correct.
iii. For projects completed in CEO-SEPAL, use map_code.
e. You can add a Display data column to enable validation on the fly. You can choose any
column from your CEO or CEO-SEPAL project. We recommend either your map class
(e.g. PL_MAP_CLASS) or your reference data class (e.g. question name column). This
example uses a CEO project.
4. Once you have set these input parameters, select Check on the left side of the window.
a. This page will simply plot your samples on a world map.
b. Fix the locations of your plots by specifying the correct columns to use as the X and Y
coordinates in the map.
c. Click the drop down menus and select the appropriate coordinate columns for X and Y
coordinates. X coordinate should be LON; Y coordinate should be LAT.
5. Next, click the Results page on the left side of the screen.
a. The Results page will display a few different accuracy statistics, including a Confusion
Matrix, Area Estimates, and a Graph of area estimates with confidence intervals.
b. The Confusion Matrix enables you to assess the agreement of the map and validation data
sets.
i. The rows represent your assignments while the columns represent the map classifier’s.
ii. The diagonal represents the number of samples that are in agreement, while the off-
diagonal cells represent points that were not mapped correctly (or potentially not
interpreted correctly).
6. Typically, you would have to create the confusion table yourself and calculate the accuracies,
however, the SAE-Analysis tool does this for you.
• You can download confusion matrix as tabular data (.csv) using the button.
7. Under Area estimates, the table shows you the area estimates, and producer’s and user’s
accuracies, all of which were calculated from the error matrix and the class areas (sample
weights) from the map product you are assessing.
a. Estimations are broken up into simple and stratified estimates, each of which has its own
confidence interval.
b. In this exercise we collected validation data using a stratified sample, so the values we
need to use are the stratified random values.
c. Note that all area estimates are in map units. For our map, this is meters.
d. You can change your desired confidence interval using the slider at the top of the panel.
e. You can Download area estimates as tabular data (.csv) using the button.
Interpretation of the accuracy estimates
The determination of a good level of accuracy for maps is not governed by a general rule.
Assessing the validity of data depends on the specific purpose of the map and must be evaluated
on a case-by-case basis. When analyzing changes in land cover, it is important to focus on the
accuracy of the change itself rather than solely relying on the accuracy of two individual land
cover maps. Even if both maps demonstrate high accuracy for a single point in time, it does not
necessarily indicate the accuracy of change classes. Instead of comparing maps from different time
periods, it is advisable to conduct a new change analysis using remote sensing images. This
approach is necessary because changes typically occur in small portions of an area and are often
smaller than the combined errors of the individual map productions (GFOI, 2013). Forest change,
for example, typically accounts for less than 1% of the total area, meaning that in two land cover
maps with an overall accuracy of 99%, the observed change could be attributed to the error
between the two maps.
Moreover, the total sample size, number of strata, and allocation of the sample size to each stratum
can influence the accuracy measures, favouring certain types of accuracy over others. Accuracy
tends to be higher for stable classes compared to change classes. Additionally, accuracy can vary
across different landscapes.
To quantify map accuracy, statistical reports such as an error matrix (also known as a confusion
matrix) are commonly used. These reports compare the map's classification with a reference
classification to assess accuracy.
Error matrix in sample counts versus Error matrix in area proportions
It is recommended to present the error matrix in terms of estimated area proportions instead of
absolute sample. Showing only the sample counts does not, for example, explicitly demonstrate
how the user’s accuracy is calculated. The estimated area proportions normalize the absolute
sample counts by the map area and are used to calculate the users and producer’s accuracy
Error matrix in
Reference
sample counts
Forest
Built-up
Agric
Waterbody
Wh
Map
Forest
90
1
10
0
0.5932
Built-up
25
10
17
0
0.0891
Agric
28
1
11
0
0.3173
Waterbody
5
4
13
8
0.0004
Total
148
16
51
8
1
Error matrix in
Reference
area proportions
Forest
Built-up
Agric
Waterbody
Wh
Map
Forest
0.52863
0.00587
0.05874
0
0.5932
Built-up
0.04283
0.01713
0.02913
0
0.0891
Agric
0.22209
0.00793
0.08725
0
0.3173
Waterbody
0.00007
0
0.00017
0.000106274
0.0004
Total
0.79362
0.03099
0.17528
0.000106274
1
V(μ)
0.0009224
0.0001216
0.0008628
1.07101E-09
SE(μ)
0.0303713
0.0110266
0.0293732
3.27262E-05
±95% CI
0.0595277
0.0216122
0.0575715
6.41434E-05
MoE [%]
7.50%
69.74%
32.84%
60.36%
User's acc.
0.891
0.192
0.275
0.267
Prod. acc.
0.666
0.553
0.498
1.000
Over. acc.
0.633
Error Types
Errors of Omission = refers to reference sites that were left out (or omitted) from the correct class
in the classified map. The real land cover type was left out or omitted from the classified map.
Error of omission is sometime also referred to as a Type I error. An error of omission in one
category will be counted as an error in commission in another category. For the forest class, is
(0.04283 + 0.22209 + 0.00007) / 0.79362 = 0.33 (33%).
Errors of Commission = the classification error, where in pixels are classified to one class but
belong to other classes in reference data. In other words, errors of commission occur when a pixel
is incorrectly included in the land use/land cover class being evaluated. Commission errors are
calculated by adding together the incorrect classifications and dividing them by the total number
of classified sites for each class. For the forest class, is
(0.00587 + 0.05874 + 0) / 0.5932 = 0.33 (0.11%).
Overall Accuracy
This is the proportion that were mapped correctly in each of the classes. The diagonal elements
represent the areas that were correctly classified.
= 0.52863 + 0.01713 + 0.08725 + 0.000106274 = 0.633 (63.3%)
This could also be expressed as an error percentage, which would be the complement of accuracy:
error + accuracy = 100%.
The overall map accuracy is not always representative of the accuracy of individual classes (GFOI,
2013).
Producer’s Accuracy
This is the map accuracy from the point of view of the map maker (the producer). It is the number
of reference sites classified accurately divided by the total number of reference sites for that class.
For Forest class, it is 67% (0.52863 / 0.79362 = 0. 0.67).
Producer's Accuracy = 100%-Omission Error.
User’s Accuracy
This is the accuracy from the point of view of a map user. It tells how often the class on the map
will actually be present on the ground. This is referred to as reliability. For Forest class, it is 89%
(0.52863 / 0.5932 = 0.89).
User's Accuracy = 100%-Commission Error
High overall map accuracy does not guarantee high accuracy a strata. Therefore, both producer’s
and user’s accuracy for all single classes need to be considered. A high user’s accuracy and low
producer’s accuracy for forest class, for example, indicate that most of the forest class in the map
was also forest class in the reference data, but that the map missed catching a fair amount of forest.
Comparing User's and Producer's Accuracy
The user and producer accuracy for any given class typically are not the same. In the above
examples the producer’s accuracy for the Forest class was 67% while the user's accuracy was 63%.
This means that even though 67% of the reference forest areas have been correctly identified as
“forest”, only 63% percent of the areas identified as “forest” in the classification were actually
forest.
Confidence Interval
The accuracy assessment is a process that helps determine the uncertainty associated with
estimates of different areas on a map. One way to quantify this uncertainty is by using confidence
intervals. A confidence interval provides a range within which the true value of an unknown
quantity is expected to fall with a specified probability. The most commonly used confidence level
is 95%, which means that there is a 95% probability that the true value lies within the given range.
The Intergovernmental Panel on Climate Change (IPCC) Good Practice Guidelines recommend
the use of 95% confidence intervals in greenhouse gas inventories. These intervals help assess the
precision of the estimates. A higher precision is indicated by smaller confidence intervals, while
lower precision is indicated by larger intervals.
In the example provided, the Forest class has the lowest precision with a confidence interval (CI)
of 0.0595. This means that with 95% confidence, the true proportion of the study area classified
as Forest is estimated to fall within the range of 0.529 ± 0.06. Similarly, the agricultural class has
a slightly higher precision with a confidence interval of 0.0576, indicating that the true proportion
of the study area classified as agricultural is estimated to fall within the range of 0.059 ± 0.05 with
95% confidence.
Documentation and archiving
Documentation of your area estimate and archiving this information for future reference are
critical in order to replicate your estimation process. Examples where you would want to repeat
your analysis include different areas (states, provinces, ecological regions) or time periods
(months, years).
Logging decision points
Logging decision points is a critical part of documenting your area estimation process and being
able to recreate your estimation process. A decision point is anywhere where you make a decision
about your project that can change the outcome. For example, “deciding that pixels with over 75%
tree cover should be classified as Forest cover” is a decision point.
Examples of points to log are
your land use decision tree; land use classification definitions; settings used for classification,
along with any refinements; dates used for change detection decisions, including what classes can
change and imagery and processes used in the classification; Stratification choices; data collection
procedures; etc.
Reporting
Writing a report summarizing your findings is a critical part of your area estimation. If you are
conducting an area estimation for internal use, this report will provide you with a blueprint for
future estimations. If you are conducting an area estimation for another organization, this report
will convey your results and the quality of your analysis. Suggested outline are:
• An introduction, describing why the project was completed and any goals of the project.
• Your project’s methods, that is how your project was completed.
• All tools used in your analysis.
• All decision points.
• Any other information needed to recreate your project.
• Your project results.
• Your area estimation.
• The error associated with your classification.
• A comparison of any self-checks (one interpreter) and cross checks (between
interpreters) with the main sets of plots.
• Any actions or next steps arising from your analysis.
Data archiving and creating metadata
For data archiving, the following information needs to be compiled:
1. A database of the sample data collected by the interpreters including:
a. The geographical coordinates in define coordinate or projection system.
b. The unique identification code for each sample unit.
c. The interpretation of all sample units including the previous interpretation(s) of the
sample unit in the case this was revised or corrected.
2. The interpretation of sample units conducted by the coordinator.
3. Metadata regarding the interpreter that collected the sample data, when the data was
collected, which data sources were used.
You have successfully completed this exercise, and now you possess the knowledge and skills
to conduct accuracy assessments and generate area estimates using SEPAL.